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ABSTRACT 


It is essential for the Marine Corps to ensure the successful supply, movement and 
maintenance of an armed force in peacetime and combat. Integral to an effective, 
long-term logistics plan is the ability to accurately forecast future requirements to sustain 
materiel readiness. Total Life Cycle Management — Assessment Tool (TLCM-AT) is a 
simulation tool combining operations, maintenance, and logistics. This exploratory 
analysis gives insight into the factors used by TLCM-AT beyond the tool’s embedded 
analytical utilities. A Java program is developed to automate multiple changes to 
TLCM-AT’s database, execute simulation runs and record output data. A scenario 
deploying LAV-25 vehicles to a tropical region, with three courses of action, provides the 
basis for analysis. The research provides a description of the analysis available by 
TLCM-AT as a stand-alone tool, and concludes with how design of experiments (DOE) 
expands insights gained. This thesis provides a framework for using DOE with 
TLCM-AT, identifies a structured use of TLCM-AT for decision makers, and provides 
enhancements that enable more effective use of TLCM-AT. Results indicate no practical 
change in operational availability (Ao) when varying five factors, using 129 design points 
and 15,480 replications. The factors adjusted are: spares, depot capacity, induction 
quantity, part repair time and part degradation time. Results also reveal synergies 


between the modeled factors and numbers of spares to be the dominant factor Ao. 
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EXECUTIVE SUMMARY 


Logistics is a vital part of the Marine Corps’ vision of developing a force capable 
of performing and successfully completing the missions of the twenty-first century. It is 
essential that the Marine Corps ensures the successful supply, movement and 
maintenance of an armed force in peacetime and combat. 

This thesis centers on the Total Life Cycle Management — Assessment Tool 
(TLCM-AT). Total Life Cycle Management (TLCM) defines the design, engineering, 
manufacturing, disassembly and disposal of a system. Decision makers who use TLCM 
must understand its impact on combat effectiveness and on the development of systems 
required for success in modern warfare. The Marines acquired TLCM-AT to improve 
upon their ability to manage TLCM and maintain a more reliable fighting force. This 
summary provides a functional overview of TLCM-AT, the research methodology, 
conclusions and recommendations. The analysis is designed to explore the capabilities of 
TLCM-AT with multiple quantitative techniques, in order to develop an analytical 
process that enables tactical users to more readily employ TLCM-AT. 

TLCM-AT is a simulation tool developed by Clockwork Solutions in Austin, 
Texas. It is a holistic, continuous-loop representation of the life cycle of a weapon 
system: combining operations, maintenance, and logistics. TLCM-AT’s model structure 
and organization enables the Marine Corps to model the myriad _ of 
industry-accepted elements that directly affect the life cycle of a system. TLCM-AT 
assists weapons systems managers with evaluating, quantifying, and reducing life-cycle 
costs without adversely impacting fleet readiness and availability. Users can run “what 
if’ scenarios by manipulating data inputs to examine the long-term effects to life-cycle 
policy decisions. Scenarios are adjusted by the user via the scenario editor in TLCM- 
AT’s graphical user interface (GUI). This research extends the normal analysis by 
implementing the concepts of data farming and design of experiments (DOE) as 
employed by the Simulation Experiments and Efficient Designs (SEED) Center at the 
Naval Postgraduate School (NPS). 
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This thesis provides the following: 


a A framework for using DOE with TLCM-AT. 
7 A structured use of TLCM-AT for USMC decision makers. 


7 Enhancements that enable more effective usage of TLCM-AT. 


A notional scenario deploying Light Armored Vehicles (LAVs) to a tropical 


environment serves as the background for this prototype analysis. According to the 


scenario, the Marine Corps seeks to deploy one LAV-25 battalion, and eventually a 


follow-on battalion for contingency operations in a hot, humid environment. 


Maintenance personnel anticipate numerous unscheduled removal events of two 


computerized parts that historically perform poorly in humid conditions. Three potential 


courses of action (COAs) are suggested to help mitigate the anticipated problem. 


COA 1: 
COA 2: 


COA 3: 


Send a large number of spares with the follow-on battalion. 


Invest in improved Line Replacement Units (LRU). The research 
and development (R&D) cost for the one-month R&D program is 
$1M, and the new LRUs cost an additional 1.5 times the cost of the 
old type. Bring a lesser number of the improved spares and install 
them whenever the old type is removed. When the old type is 
condemned (removed from maintenance system), buy the 
new type. 


A variant of COA 2, except that the old type is purchased 
whenever the LRUs are condemned. No money is invested in the 
new LRUs. The idea is to save some money, while accomplishing 
the same goals. 


Each scenario is modeled separately in TLCM-AT’s Microsoft Access database. 


Five factors are chosen to explore over the COAs using Nearly Orthogonal Latin 


Hypercube (NOLH) experimental designs. A Java program is created to automatically 


run each scenario using 129 input combinations. The five factors varied in this study are: 


7 Spare levels: The total number of spares at each repair location. 


7 Induction Quantity: A limit on the number of inductions (number of 
items a maintenance facility can accept) that can occur in the given quarter 
and year at a single repair facility. 


7 Capacity: The number of parts that can be processed concurrently at a 
single repair facility. 
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7 Service Times: The time it takes to repair a part. 


7 Unscheduled Removal Rates: The part failure rate. 


The NOLH allows for systematic variation of the factors to broadly explore their 
effect on the measure of effectiveness (MOE). Ao is chosen as the MOE for this study. 
Upon completion of the Java program, a comma-separated file is created containing Ao 
by quarter over a 20-quarter period. 

First, each scenario is run once with 100 replications. This type of simulation 
typifies the kind of study a common user of TLCM-AT would use to produce a 
quick-turn analysis of a specific scenario. Results show COA-2 saves $3.2M compared 
to COA-1 and $15.8M compared to COA-3. COA-2 had lower overall operation and 
sustainment (O&S) costs for spares replenishment and produced the least impact on 
repair facilities by producing a lower number of maintenance tasks performed. 
Furthermore, each scenario maintained relatively equal Ao throughout the contingency 
and for the full 20-quarter period. All scenarios produced a much higher Ao than the 
baseline, thus maintaining the status quo. 

DOE implementation with TLCM-AT across all scenarios and 129 design points 
each produced similar results in regard to Ao. Statistically significant differences in Ao 
are observed after quarter 12 (the end of contingency operations). However, these 
differences are not practically significant, as Ao varied no more than 4% for 
each scenario. 

Regression analysis produced relatively low R* values, representing a low amount 
of variance explained in the model by the five factors. In cases where R’ is low, the 
results show that other factors (besides those used in this study) should be considered. 
Thirty-two stepwise regressions are conducted, and R’ increases in all cases as interaction 
terms are added to the models. This result shows that including main effects alone did 
not capture the complete picture. Significant factors for scenario two are determined to 
be spares, spares x spares, spares x service time and induction quantity x degradation 
time. The significance of interaction terms reveals synergies between the factors. 

Paired two-sample t-Tests between each scenario are completed to determine 


whether or not two population means of Ao are equal. In each test, the hypothesized 
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mean difference is zero. Low p-values are produced for all six comparisons, revealing a 
statistically significant difference in mean Ao between each scenario. However, the 
mean difference is not practically significant because the variation in mean Ao is small. 
For example, going from 81.8% to 82.1% Ao is not significant enough to make practical 
impacts on the overall operational picture over a range of inputs. However, there are 
some differences in other measures such as cost and maintenance utilization. 

In conclusion, exploitation of TLCM-AT is accomplished through development of 
a Java program to automate implementation of DOE and NOLH with TLCM-AT. The 
simulated scenarios used in this analysis form a strong foundation for further TLCM-AT 
studies that use DOE to analyze life cycles. Java code written to extract many different 
MOEs gives insight to the factors most affecting outcomes of simulation runs. This 
analysis adds depth to typical “what if’ scenario runs, and informs decision makers on 
the consequences of decisions regarding Marine Corps weapon systems. It is 
recommended that research continue into embedding TLCM-AT in a data farming 


environment. Furthermore, it is recommended that a prototype study be performed. 
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I. INTRODUCTION 


Develop better readiness and sustainment indicators based on _ predictive 
modeling, so that timely changes to strategies, plans, and programs can 
be implemented. 


Commandant’s Planning Guidance, November 2006 


A. BACKGROUND AND MOTIVATION 


The United States Marine Corps (USMC) continually works to shape logistics 
plans and policies to sustain excellence in combat effectiveness. Logistics is an 
extremely vital part of the Marine Corps’ vision of developing a force capable of 
performing and successfully completing the missions of the twenty-first century. It is 
essential that the Marine Corps ensures the successful supply, movement and 
maintenance of an armed force, both in peacetime and in combat. Integral to an effective, 
long-term, logistics plan is the ability to accurately forecast future requirements to sustain 
materiel readiness as well as determine future force structural needs. This study focuses 
on the elements that affect the life cycle of weapons systems, such as level of spares, 
depot capacity and repair times. 

Enterprise-level Total Life Cycle Management (TLCM) is the formal process to 
identify, analyze, and implement synergistic “cradle-to-grave” solutions that optimize the 
acquisition and logistics chain across the Marine Corps. The following items impact the 


entire Marine Corps and require an enterprise view!: 


. Automatic Logistics/Prognostics 

7 Performance-Based Logistics 

7 Design-in Reliability, Maintainability and Supportability 
7 Direct Vendor Delivery 

7 Logistics Footprint 


7 Fuel Efficiency 


! Marine Corps Order 4000.57, Enclosure (1), Commandant of the Marine Corps, Subject: Marine 
Corps Life Cycle Management, 16 September 2005, p. 1. 
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id Depot Maintenance 


7 Condition-Based Maintenance Technology 
. Logistics Operational Architecture 
a Automated Identification Technology 


TLCM defines the design, engineering, manufacturing, disassembly and disposal 
of a system. TLCM includes the set of decisions and actions that determine the 
performance and availability of a weapons system in the context of its environment. 
Therefore, decision makers must use TLCM and understand its impact on combat 
effectiveness and on the development and cost of weapons systems needed for success in 
modern warfare. 

The Marine Corps has invested in Clockwork Solutions (www.clockwork- 
solutions.com) to adapt a discrete event simulation tool for use in conducting TLCM. 
Clockwork Solutions is a provider of reliability-centered total life cycle performance 
prediction tools. One of their simulation tools, the TLCM Assessment Tool (AT), is an 
emerging enhancement module to the Life Cycle Modeling Integrator (LCMI). LCMI is 
a Web-based set of decision support modules that integrates historical data and converts 
it to quality logistics intelligence for the USMC. Within the TLCM-AT module is the 
Fault Isolation (FI) model, which uses a multivariate, dimension reduction algorithm to 
create a ranked index of the components of a principal end item (PEI). Program 
Managers (PMs) can use this decision support tool to enhance the effectiveness of their 
depot maintenance rotation programs. The FI model can also help identify components 
to be inspected during Inspect and Repair Only As Necessary IROAN) programs. 

The aging of such systems as the Amphibious Assault Vehicle (AAV) and the 
Light Armored Vehicle (LAV) creates a greater level of uncertainty about their Ao. 
Simulation tools, such as TLCM-AT, are helpful in identifying critical factors that affect 
the Ao of weapon systems. Simulation can make forecasting more accurate and enable 
decision makers to be proactive with TLCM. An exploratory analysis on model factors 
pinpoints factors with the strongest effects on output such as system reliability, Ao and 
cost. Furthermore, sensitivity analysis of critical factors better defines thresholds for 


decision makers. This thesis examines the capabilities of TLCM-AT. 
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B. RESEARCH QUESTIONS 


The exploratory analysis gives insight into the factors used by TLCM-AT. While 


this analysis is not exhaustive, the following questions guide this research: 


7 Which factors modeled in the TLCM-AT have a critical effect on the Ao 
of the LAV-25? 


7 How should data farming be applied to the TLCM-AT model? 


7 How sensitive are the factors (spares, capacity, induction, service time, 
degradation time) to defining thresholds for decision makers? 


Cc. SCOPE OF THESIS 


This analysis examines the capabilities of the TLCM-AT. The goal is to use 
multiple quantitative analysis techniques in order to develop a methodology to enable 


tactical users to more readily employ the TLCM-AT tool. 


D. METHODOLOGY 


This thesis uses the TLCM-AT, for which three scenarios are created, each 
revolving around a potential decision about the LAV-25. Each scenario represents a 
possible course of action (COA) for the decision maker. In each scenario, five factors 


are adjusted: 


7 Spare levels: The total number of spares at each repair location. 

7 Induction Quantity: A limit on the number of inductions that can occur 
in the given quarter and year at a single repair facility. 

7 Capacity: The number of parts that can be processed concurrently at a 
single repair facility. 

7 Service Times: The time it takes to repair a part. 

7 Unscheduled Removal Rates: The part failure rate. 


This study determines the effect these factors have on the Ao of the LAV-25. Ao 
is defined as the number of operational platforms divided by the total number of 
platforms at a given base or bases at a given future time interval. Each scenario is 


explained in detail in Chapter III. 


The scenarios are developed based on a range of inputs of the factors listed above. 
The factors are used to create several design points developed using the 
Nearly Orthogonal Latin Hypercube (NOLH).? Scenario runs are replicated in the 
simulation and then analyzed. The analysis uses data farming, as employed by the 
Simulation Experiments and Efficient Designs (SEED) Center at the Naval Postgraduate 
School (NPS) and the International Data Farming Workshop. Data farming uses 
high-speed computing to run simulations thousands of times while simultaneously 
varying many input factors. Using experimental designs developed at NPS, the data 
farming results is analyzed. The statistical analysis reveals the strengths, weaknesses and 
sensitivities of the model. The result is a preliminary study of support, based on analysis, 


to aid future users of TLCM-AT. 


E. BENEFITS OF THE STUDY 


This study provides the Marine Corps with analytical support for TLCM-AT. 
Data farming and NOLH provide a unique approach for the study by allowing the 
identified factors to be widely varied, and for multiple simulation runs to be 
accomplished for post-run analysis. Chapter II provides a brief description of discrete 
event simulation modeling followed by a functional overview of TLCM-AT. Chapter III 
explains specifically how data farming and NOLH are used in the simulation runs. 
Chapter IV contains data analysis from the resulting data obtained from multiple 
simulation runs. Chapter V provides conclusions and identifies future areas of study. 
Currently, the Marine Corps does not know all the capabilities and limitations of 
TLCM-AT. The aim of this study is to provide those insights into how TLCM-AT can be 


used to help decision makers make more informed decisions regarding TLCM. 





2 NOLH concept developed by Lieutenant Colonel Thomas M. Cioppa, United States Army. Detailed 
description found in his dissertation, “Efficient Nearly Orthogonal and Space-Filling Experimental Designs 
for High-Dimensional Complex Models,” Ph.D. in Operations Research, Naval Postgraduate School, 
Monterey, California, September 2002. 
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I. MODEL BACKGROUND AND CAPABILITIES 


A. INTRODUCTION 


Before conducting an exploratory analysis, one must understand the intricacies of 
the model itself. The purpose of this chapter is to introduce the reader to TLCM-AT. It 
begins with a brief description of mathematical models, both deterministic and stochastic. 
This discussion leads into a functional overview of TLCM-AT, including criteria, 


structure and applications. 


B. DISCRETE EVENT STOCHASTIC MODELING 


Mathematical models can be categorized broadly as probabilistic or deterministic. 
A deterministic model uses factors and variables without random fluctuations. The 
system is entirely defined by the initial conditions. A deterministic model provides a 
single point estimate which serves as a “best guess” for an unknown population 
parameter. A stochastic model considers randomness in one or more of its factors or 
variables. By allowing random variation in one or more inputs over time, a stochastic 
model estimates probability distributions of potential outcomes. What results is not a 
single point estimate but a distribution of possible outcomes. 

After empirical data is collected, a theoretical probability distribution is often fit 
to the data to attain a more general description of the underlying process. The population 
parameters can be approximated for the distribution. Stochastic processes provide a 
better understanding of inherently stochastic real-life situations. 

The TLCM-AT is a discrete event stochastic model. The aim of TLCM-AT is the 
integration of discrete event performance models into the USMC decision support 


enterprise. Clockwork Solutions has modeled five USMC systems: 


a Light Armored Vehicle (LAV-25) 

. Medium Tactical Vehicle Replacement (MTVR) 
7 Lightweight 155mm Howitzer (LW 155) 

7 Joint Light Tactical Vehicle (JLTV) 


. Amphibious Assault Vehicle (AAV) 
5 


These five systems are chosen based on the following criteria: 


7 Maintenance Cost 
7 Operational Usage 
7 High and Low Readiness 


7 Sample Size (High and Low Density Table of Authorized Material Control 
Number [TAMCN]) 


. Old System 


7 New System 
a Recent Mission Change 
" Configuration Change 


Cc: TLCM-AT FUNCTIONAL OVERVIEW 


TLCM-AT is a technology developed to assist weapon systems fleet managers 
with evaluating, quantifying and reducing life-cycle costs without adversely impacting 
fleet readiness and availability.3 This model is a holistic, continuous-loop representation 
of the life cycle of any weapons system: it combines operations, maintenance, and 
logistics, as shown in Figure 1. The inner loop contains examples of the inputs to 
TLCM-AT, and the items on the outside of the loop represent examples of outputs 


obtained after simulation runs. 


3 Clockwork Solutions, Inc., “Predictive Modeling Technology and Consulting Services in Support of 
Phase III TLCSM,” Final Project Report (1992-2005), submitted to Headquarters, Marine Corps 
Installations and Logistics, 31 December 2007, p. 1. 
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Figure 1. TLCM-AT continuous-loop model (After)* 


An individual weapons system TLCM-AT model is composed of six 


interactive components?: 


7 Initialization: Readiness condition and location of systems and parts at 
simulation start. 

7 System Model: Work breakdown structure of the system and its variants. 

7 Operations Model: Current and future operations according to base 


location, platform type, or serialized system. 


7 Maintenance Model: Actions on a component in maintenance as 
described by capacity constraints. 


7 Sustainment Model: Spares, lateral resupply, depot upgrades, and 
induction programs. 


7 Cost Model: Cost of purchases and activities, including maintenance, 
training, initial and subsequent provisioning of parts, storage, shipping, 
and upgrades. 





4 Clockwork Solutions, Inc., “TLCM-AT Training Material,” Chapter 2, Naaman Gurvitz, Ph.D. 
7 August 2007, p. 2. 


5 Clockwork Solutions, Inc., Technical Reference Manual, “Aircraft Total Life Cycle Assessment 
Software Tool (ATLAST),” Version 5.0., (1992-2005), Austin, Texas, p. 14. 
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The system model consists of input data that represents the structure, location, age 
and status of components (installed, spares, serviceable/unserviceable). The System 
Operational Effectiveness (SOE) parts usage database is used to build the system 
structure for the AAV, LAV, and MTVR models. SOE is a tool developed by 
Marine Corps Systems Command to monitor and measure system operational 
effectiveness attributes on a wide range of Marine Corps ground weapons systems. It 
automatically calculates and summarizes key reliability, maintainability and availability 
metrics. SOE provided part usage data as well as failure rates. The rest of the models are 
built from a variety of sources: spreadsheets from Marine Corps Logistics Command 
(LOGCOM), the Capability Assessment Support Center (CASC), and queries to the 
program manager’s office for the individual weapon systems. The LW-155 model does 
not use SOE at all; it starts with original equipment manufacturer (OEM) structure and 
reliability information. Lastly, the JLTV model is built by extracting relevant subsystems 
from a high-fidelity LAV model with an indentured system structure. Failure rates for 
the JLTV components are exponentially distributed and in accordance with historical 
LAV maintenance records. For an exhaustive list and explanation of TLCM-AT’s inputs 
and outputs, refer to the technical manual, which can be _ provided by 
Clockwork Solutions. 

The operations module consists of the base structure and placement of systems 
and the current and future usage rates. Figure 2 shows a mapping of the base structure 
and system placement for the LAV variants. The MTVR model has a total of 7,597 
vehicles distributed between I-[V Marine Expeditionary Force (MEF), 7 MEF, Maritime 
Prepositioning Ships (MPS) and Depot inventory. The JLTV model uses eight notional 
bases and will gradually field 900 systems among them from 2009-2012. The LW-155 
model places the Marine artillery regiments at the Operational Level (O-level), in 
addition to the Enhanced Equipment Allowance Pool (EEAP), the Army’s field artillery 
school at Fort Sill, and the Aberdeen Proving Grounds. Guns are fielded according to the 


schedule in the User’s Logistics Support Summary (ULSS). 
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Figure 2. Base structure and system placement in the LAV model 


Operational tempos for baseline models are set to USMC fleet averages. 
Deployed units (7 MEF) have roughly five times the usage rate as nondeployed units. 
Units in stores or MPS are not operated. Op-tempo for the LW-155 is assumed to be 
1,000 km driven and 100 rounds fired per year for all each. 

The maintenance component determines the task times and effects on the system 
and its subsystems upon entering maintenance actions. TLCM-AT uses part numbers 
(if applicable) to model preventive maintenance, partial repairs, depot upgrades, principle 
end-item (PEI) depot rotations, mandatory life limited removals, and 
unscheduled failures. 

For the SOE-based AAV, LAV, and MTVR models, task times are taken from 
maintenance allocation charts (MAC) which are estimated by maintenance personnel. 


Task times are approximated as follows: 


7 Tear Times: Fifteen minutes for consumable items; 150 minutes for 
repairable items. 


7 Default Repair Times: Fifty days for depot level; 3.65 days for 
intermediate and operational level. 


. Other Repair Times: The repair cycle time (RCT) from the 
Master Work Schedule (MWS) where applicable. 


7 Build Times: Only apply to AAV model where a one-day system build 
time is used to represent the limited technical inspection (LTI) process at 
the conclusion of maintenance. 


. Inspection Times: Constant; the AAV model has a one-day system 
inspection time to represent problem diagnosis. 


The maintenance task times in the LW-155 and JLTV models are of much higher 
fidelity. Inspection, tear, repair, and build times are set according to part or subsystem 
using both measured and estimated crew times (from LAV subsystems in the case of 
the JLTV). 

Depot capacity constraints in the AAV, LAV, and MTVR models are determined 
according to the actual depot MWS for Fiscal Years (FYs) 2007-2009. For 
FYs 2010-2011, the 2007-2008 schedules are repeated. These are adjusted according to 
fleet needs. Deployed units have no depot capacity constraints; all necessary work is 
assumed to be completed. I-Level capacity constraints are set proportionally to the 
number of mechanics in the maintenance battalion table of organization (T/O): 80% for 
nondeployed units and 100% for deployed units. In the LW-155 and JLTV models, there 
are currently no capacity constraints. 

In the sustainment component, a “buy upon condemnation” feature and USMC 
spare inventories can be controlled from the TLCM-AT analysis control panel. USMC 
spare inventories are apportioned in the AAV, MTVR, and LAV models as scenarios. 
The baseline models, instead, include an optimized spares package. The LW-155 and 
JLTV models also use optimized spares packages. None of the models currently feature 
any depot upgrade programs or PEI rotations, although the tool provides this capability. 

Shipping-time distributions for the AAV, MTVR, and LAV models are fit to 
historical data. Many fits are lognormal. The LW-155 and JLTV models use appropriate 


constant times for shipping between maintenance levels. 
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In the cost component, TLCM-AT enables computation of the following total 


life-cycle costs: 


. Initial provisioning 

7 Spares storage 

7 Spares replenishment 

7 Shipping 

a Supply administration 

. Manpower 

" Training 

7 Test equipment maintenance 
7 Test equipment space 

s Development 


Costs may be displayed as a function of time, as totals, and per operating hour. 
Furthermore, costs related to individual Line Replaceable Units (LRUs) may be analyzed. 


TLCM-AT features include: 


7 Uses Marine Corps’ automated SOE Decision Support Tool. 

7 Database structure provides for plug-ins and simulation of numerous 
systems and platforms. 

s Provides “what if’ simulation scenario management for analysis of 
life-cycle sustainment forecasts. 

7 Includes operation and maintenance models. 

a Forecasts by part numbers or serial numbers by equipment group. 

a Maintenance modeling—phased inductions include three options: 
scheduled or preventative, causal, or opportunistic. 

7 Life-cycle impact assessments include three aging options: aging by 
assembly, location, and repair interval. 

" Ability to initialize system state, component state, and life-cycle 
metric prediction. 

a Operational variations by location. 

7 Part interchangeability and substitutability rule manipulation. 

a Repair capacity evaluation, which allows for measuring total repair cycle 


time for individual LRUs. 


a Cycle time evaluation (repair, transportation, and order lead time). 
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id Up to three maintenance levels. 


7 Time-dependent forecasting, which allows for any model outputs to be 
displayed in time-based format. 


With the TLCM-AT tool, users may quantify life-cycle costs and impacts due to 
management decisions regarding equipment configuration changes, remaining-life rules 
modifications, alternate sparing strategies, adjusted operating hours programs, modified 


repair concept, and age and reliability factors. Applications for use include: 


" Maintenance concept modifications. 

7 Deployment strategy assessments (operations). 

7 Systems degraders’ assessment (sensitivity). 

7 Reliability modifications impacts (build and configure). 


7 Supply chain alternatives (supply, stock, and resupply). 


7 PEI rotations (fleet management). 


Military services use a complex system of maintenance and supply to manage 
reparable items in their inventories using two key concepts: level of repair 
and indentures.® 

Levels of repair are differentiated by their varying the number of resources and 
capabilities. Organizational (or operational) repairs are closest to the weapons system 
and consist mainly of diagnostics and remove-and-replace maintenance of the weapons 
system itself. Depot-level maintenance exists farthest from the weapons system. 
Depot-level maintenance includes diagnostics, overhaul, and remanufacturing. One or 
more levels of intermediate repair may exist between organizational and depot-level 
maintenance. TLCM-AT models three levels of repair: operational, intermediate 
and depot. 

Reparable items consist of smaller subcomponents, forming an indenture 
relationship between parent parts and child parts. Having an_ indenture-level 
organizational structure means an item can be repaired at some intermediate level by 
removing and repairing components and subcomponents. The indenture relationship 

6 Bradley E. Anderson,. Marvin A..Arostegui, and David L. Lyle, “Methods for Conducting Military 


Operations Analysis,” Military Operations Research Society, LMI Research Institute, McLean, Virginia, 
2007, p. 352. 
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requires logisticians and supply personnel to balance stock inventories to stock the right 
expensive items and smaller, cheaper subcomponents. These items must also be placed 
at the right level of repair. The TLCM-AT model uses the indenture-level concept. 


Figure 3 shows the basic structure of the indenture levels modeled in TLCM-AT. 


Indenture Levels 
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Figure 3. TLCM-AT indenture structure 


The TLCM-AT model’s structure and organization enables the Marine Corps to 
model the myriad of industry-accepted elements that directly affect the Ao of a system, 
and easily run “what if’ scenarios by manipulating data inputs to examine the long-term 
effects to life-cycle elements. _TLCM-AT provides a common tool across the 
Marine Corps to meet enterprise-level concerns and identify where improvements can be 
made with regard to process and policy, asset management and performance-based 


logistics (PBL) validation. 
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Hl. SCENARIO DEVELOPMENT AND DATA GENERATION 


A. INTRODUCTION 


This chapter explains the scenario and the data generation for use in the post 
experiment analysis. The concepts of DOE and NOLHs aid in the analysis of a complex 
model like TLCM-AT. Data is generated and automated to support the analysis. A 


fictional LAV scenario is used as the basis for the simulation. 


B. DESIGN OF EXPERIMENTS 


TLCM includes everything from the design, engineering, and manufacturing, to 
the disassembly and disposal of a system. Life-cycle simulation tools are complex and 
contain many factors or variables. To identify robust mixes of these factors and their 
synergies, Over many scenarios, experimental designs are needed. The DOE concept 
allows researchers to simultaneously vary the levels of factors (or inputs), resulting in an 
estimate of factor effects on response variables (outputs). 

To understand the contributions of individual factors, each factor is varied to 
produce all possible combinations. This is sometimes referred to as a full-factorial 
design, which considers each possible factor combination. The number of simulation 
runs required in a full-factorial design increases exponentially with the number of factors. 
For example, a full-factorial design of 12 factors, each having ten levels, requires 10'* 
(one trillion) runs. Running this many combinations is impractical. To avoid this 
difficulty, a base case scenario may be compared to others by changing one factor at a 
time. While this is a time-saving approach, estimates of interactions (synergies) among 
the different factors may be overlooked, and conclusions may be inaccurate. Fortunately, 
these difficulties can be mitigated by applying the NOLH. 

NOLHs allow exploitation of a large portion of the factor space without requiring 
an unrealistic number of runs. These designs are nearly orthogonal because a small 
degree of nonorthogonality is allowed in order to achieve better space-filling. A design 


matrix is referred to as nearly orthogonal if the maximum absolute pair-wise correlation 


15 


between any two columns is less than .05. The small sample of the NOLH is seen in 


Table 1, and the full NOLH can be found in the Appendix. 


low level 
high level 
decimals 
factor name 





Table 1. Spreadsheet containing a small portion of the NOLH for this thesis 


Each row in the yellow-shaded section in Table 1 represents one design point or 
simulation run (Spares = number of spares, IQ = induction quantity, I Cap = I-level 
capacity, Deg = degradation time, ST = service time). The number under each column 
represents the level of that factor during that particular design point’s simulation run. 
This analysis required 129 rows (design points) in the yellow-shaded area. After all 
design points are replicated, the effects of each factor and factor combination can be 
analyzed. Figure 4 shows all two-way input combinations. Each cell shows all input 


combinations between the factors on the x-axis and the factors on the y-axis. 
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Figure 4. All two-way input combinations 


C. DATA GENERATION AND FLOW 


This section briefly describes the data generation. TLCM-AT uses a Microsoft 
Access database for both inputs and outputs (both are contained in the same file). 
Varying levels of multiple factors over many design points can be tedious, time- 
consuming, and virtually impossible for large-scale changes. To alleviate this problem, a 
procedure is developed to efficiently change the levels of multiple factors in the database, 
send them to the TLCM-AT for a simulation run, and gather the results. This procedure 


uses a Java program. A representation of the data generation is provided in Figure 5. 
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Figure 5. Diagram of data generation process 


The Java program receives the NOLH design matrix and the baseline database for 
each scenario. The program reads the first design point row and the levels for each factor 
and then goes into the database to adjust each factor in the appropriate input table. Next, 
the updated database is sent to TLCM-AT for a simulation run on that single design 
point. The output data generated from the simulation run is then collected by the 
program and written to a comma-separated-file. This procedure is repeated automatically 
for each design point. Upon completion of all simulation runs, a single file containing 
the data of interest for each design point is available for post-analysis. Detailed code for 
this process can be found in an NPS thesis scheduled for completion in September 2008, 


by Lieutenant Commander Alberto A. Garcia, United States Navy. 


D. SCENARIO DEVELOPMENT 


This section explains the scenarios used for this thesis. These scenarios represent 
possible COAs for a notional contingency deployment for the LAV. Each scenario is 


modeled separately in four different databases to represent the particular plan for the 
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replacement of two trouble parts. The scenarios were modeled by a modeler and analyst 


from Clockwork Solutions (Dr. Peter Figliozzi). 


1. Notional Background 


The Marine Corps is given a warning order to rapidly deploy for combat 
operations in a tropical region. The mission details and terrain drive the Marine Corps to 
include at least a battalion of LAVs in the force mix. The deployment consists of a lead 
LAV battalion and a follow-on LAV battalion. 

From historical data of combat operations in hot, humid environments, LAV 
maintainers anticipate numerous unscheduled removals rates of two computerized LRUs 


on the LAV weapons system: 


7 OT 702275001, SENSOR UNIT, LASER DESIGNATOR 
7 OT 702261001, CONTROL DISPLAY UNIT 


The problem with the computerized LRUs first becomes apparent only a few 
weeks after the lead battalion’s arrival in theater. This maintenance problem is a 
detriment to the Marines’ ability to accomplish their mission. Given time to assess the 


situation, three COAs are suggested: 


COA 1: Send a large number of spares with the follow-on battalion. 


COA 2: Invest in improved LRUs. The research and development (R&D) 
cost for the one-month R&D program is $1M, and the new LRUs 
cost an additional 1.5 times the cost of the old type. Bring a lesser 
number of the improved spares and install them whenever the old 
type is removed. When the old type is condemned, buy the 
new type. 


COA 3: A variant of COA 2, except that the old type is purchased 
whenever the LRUs are condemned. No money is invested in the 
new LRUs. The idea is to save some money, while maintaining 
the same level of Ao and maintenance utilization. 


2 Simulation Scenarios 


This subsection provides some technical details of each of how the scenarios 
(or COAs) are modeled in the TLCM-AT database. Four scenarios are used. The first is 
a baseline scenario with no corrective action taken (maintaining the status quo). 
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Scenarios 1-3 will be compared against the baseline scenario. A Stock Record Account 
Number (SRAN) is used to indicate the location of the LAV. For example, SRAN 10001 
indicates that the LAV is located with I-MEF, and SRAN 10007 indicates that the LAV is 
located with a deployed MEF. Each scenario is detailed below: 


LAV 3042 v5 Tropical Baseline Scenario: 


This is the baseline scenario that maintains the status quo: 


7 Baseline model is LAV 3042 v5 (SOE based, LRU-only model). 


7 SRAN 10007 LAVs are turned off (zero optempo) except during the 
following times: 2008 Q4 — 2010 Q1 inclusive (the contingency period). 


7 The optempo for all SRAN 10007 LAVs during the contingency period is 
2,000 hours/year. 


7 An initial force of 88 LAVs (including 45 LAV-25s) at SRAN 10007 
operates throughout the contingency period. 


7 All of SRAN 10001 LAVs (111 total LAVs including 51 LAV-25s) 
redeploy to SRAN 10007 in 2009 Q1. In 2010 Q1, they redeploy back to 
SRAN 10001. Thus, they are active in SRAN 10007, together with the 
initial force, from 2009 Q1 — Q4 inclusive. 


7 The failure rate for OT 702275001, SENSOR UNIT, LASER 
DESIGNATOR, is set to 4.3 per 10,000 operating hours, only at 
SRAN 10007. Elsewhere, it is 0.43. 


7 The failure rate for OT 702261001, CONTROL DISPLAY UNIT, is set to 
4.2 per 10,000 operating hours, only at SRAN 10007. Elsewhere, it 
is 0.42. 


LAV 3042 v5 Tropical — Scenario 1: 
This scenario represents the plan to bring old-type spares with the 


follow-on battalion: 


7 Add numerous amounts of spares to the deployment. 


. Quantity of 100 of the two troubling LRUs are added to SRAN 10007 as 
spares, during 2009 Q1. 


LAV 3042 v5 Tropical — Scenario 2: 


This scenario includes investment in improved SENSOR UNIT and CONTROL 
DISPLAY UNIT; however, limited quantities are immediately available (50 of each for 
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the deployment). They are taken on deployment as spares. Additionally, when the old 
types are condemned, the new type is bought as replacements. 
7 Added OT 7022275002 and 702261002 to *Object type table (duplicates 


of 001 versions, added a “2” to the end of the part number to differentiate 
it from the old part). 


7 Changed *Preferred buys table for 001 type to 002 type. 
7 Added 002 type to *Preferred buys table. 
7 Duplicated *Server times table for 002 types (made same as 001 types). 


7 Changed *Unscheduled removal rates table to 0.4 for both 002 types 
(all locations). 


7 Changed *Slots by type table for LAV-25 to prefer 002 variant. 

7 Made price for type 002 in *LCC costs table 1.5 times the 001 price. 

7 Quantity of 50 of the new type (002) of the two trouble LRUs added to 
SRAN 10007 as spares, 2009 Q1. 


LAV 3042 v5 Tropical — Scenario 3: 


This scenario is the same as COA 2, except the initial purchase of 50 new 002 
types occurs only once. No new types are ever purchased upon condemnation (only the 


001 types are purchased, whenever a 001 or 002 type is condemned). 


7 Copied COA 2 database to create COA 3 database. 
7 Changed *Preferred buys table for 001 type back to 001 type. 
7 Changed *Preferred buys table for 002 type to 001 type. 


E. SIMULATION RUNS 


Five factors are varied during each scenario. The levels of each factor are 
summarized in Table 2: 


HIGH LOW 
FACTOR RANGE RANGE DESCRIPTION 


Spares Spares level set to the value in the NOLH column 


Induction Quantity Induction Quantity set to the value in the NOLH column 
I-Level Capacity 1 Capacity at the I-Level set to the value in the NOLH column 
Degradation Rate 0.5 ‘ Unscheduled Removal Rate multiplied by the value in the NOLH column 
Service Time Service Time (time to repair part) multiplied by the value in the NOLH column 





Table 2. Range of changes for the factors studied. 
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The level of the five factors is adjusted for 25 LRUs, which are chosen as the top 
25 degraders. They are the 25 parts that cause the most problems during the life-cycle of 
the LAV-25 in the baseline model. The top degraders are ranked using a formula 
provided in the final project report issued by Clockwork Solutions on 31 December 2007: 


Score = (Waiting time) * (Requests) * (Unavailability +1) 


Waiting time is the average logistics response time. Requests is the number of times the 
part is requested by the fleet. Unavailability is the fraction of part-needed-but-not-spared 
occurrences by location. Unity (+1) is added to Unavailability so that parts with frequent 
failures are appropriately spared. 

The primary measure of effectiveness (MOE) is Ao. After each simulation run, 
Ao is gathered for each quarter, for a period of five years from the *out Availability 
output table from the TLCM-AT. (It should be noted that the *out Availability table 
gives the Ao as the mean of the number of replications.) 

Each of the four scenarios (the baseline plus the three COAs) is simulated in 
TLCM-AT using the DOE concept and the NOLH, as previously discussed. For each 
scenario, the 129 design points are formed from the same NOLH design matrix. For each 
design point, 30 replications are completed. The constraint on the number of replications 


is run time. Each design point takes approximately 30 minutes to run. In summary: 


7 Total design points: 4 scenarios with 129 designs point each = 516 

" Total replications: 129 design points with 30 replications = 15,480 

a Total time for all simulation runs: 516 design points for 30 minutes each 
= 258 hours (10+ days). 

7 Simulation runs executed on two standard desktop processors for a total 


duration of approximately five days. 
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IV. ANALYSIS 


A. INTRODUCTION 


This chapter describes the data collection and post processing. In accordance 
with the scenarios described in Chapter II, the analysis centers on evaluation of Ao at the 
end of quarters 12 and 20: quarter 12 is the end of the forecasted end of the contingency, 
and quarter 20 is the end of the five-year period of evaluation (five quarters of 
high-tempo operation and normal operation for the remaining time). Section B contains 
an interpretation of the results produced by TLCM-AT after running each scenario, and a 
discussion of the conclusions that may be drawn from the results. Section C gives 
detailed analysis and discussion of the results produced after simulation runs with DOE 
implementation. Throughout the analysis, insight into the TLCM process is more 
important than specific numerical results. The focus of this chapter is to show how 
TLCM-AT, combined with DOE, can be used by analysts to maximize the insight drawn 


from discrete event simulation. 


B. TLCM-AT RESULTS 


The analysis begins by examining TLCM-AT as a stand-alone tool, without 
implementation of DOE. Each scenario run is simulated through TLCM-AT using 
100 replications. At the completion of the four simulation runs (one for each COA and 
one for the baseline), post-processing applied with TLCM-AT’s graphical-user-interface 
(GUID) and its built-in analytical utilities. The TLCM-AT graphs are best viewed in color. 

In an example of quick-turn analysis using TLCM-AT, a cost comparison is first 
completed. Results from each COA, as they relate to cost, are depicted in Figures 6 


and 7. 
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Time Dependent O&S Cost Graph’ 
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Figure 6. Operation and Sustainment (O&S) cost graph 
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Overall Total O&S Cost Graph’ 
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Figure 7. Overall total O&S graph 


Figure 6 represents time-dependent O&S costs for spares replenishment for each 
COA over the five-year period. The peak evident in the graph indicates the time period 
in which the contingency operations are being conducted. Figure 7 depicts the overall 
total O&S cost for each COA. COA 2 saves $3.2 million compared to COA 3 and 
$15.8 million compared to COA 1. The results quantify the immediate value of the 
engineering change program (ECP) to improve the two trouble parts. Even though the 
price of improved parts is 50% greater than the price of the old parts, the ECP immunizes 
the LAV-25 fleet against humidity problems in the future and saves money in 


the meantime. 


Zo 


Next, a comparison is done to reflect how each COA affects Ao. This comparison 
includes the baseline scenario, representing the decision to maintain the status quo. 


Figure 8 depicts time-dependent Ao of the LAV-25 for each COA. 


Time Dependent Availability by System Type(s), 


System Type E0947 with Base Base 7 - Jungle from Baseline(Status quo) 
System Type E0947 with Base Base 7 - Jungle from COA1 

System Type E0947 with Base Base 7 - Jungle from COA2 

System Type E0947 with Base Base 7 - Jungle from COA, 3 


20 
10/7 2006 10/7 2007 10/7 2008 10/7/2009 10/7/2010 77/2011 


Date 





Figure 8. Time-dependent Ao 
As Figure 7 reveals, each COA (1-3) maintains relatively the same Ao throughout 


the contingency timeframe, as well as throughout the full 20-quarter period. Also, 


maintaining the status quo will greatly reduce Ao during the contingency, as depicted 
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with the red (lowest) line in Figure 7. In regard to Ao, it is evident that taking some 
action, whether it is COA 1, 2, or 3, is much more preferable to maintaining the 
status quo. 

Adding more old spares or introducing new spares into the maintenance system 
will undoubtedly have an effect on a repair facility. TLCM-AT can display this effect by 
examining the number of maintenance tasks performed during the contingency on the 
laser designator unit. COA 2 clearly produces the lightest workload for the maintenance 
facility, as depicted by the blue (lowest) line in Figure 9. Base Shop 7 in Figure 9 
represents the repair facility designated for maintenance on the LAV-25s during 


the contingency. 


Time Dependent Tasks Performed 


Repair for WWUC LASER DESIG SENSOR UNIT with Base Shop 7 - Jungle Int from COA1 
Repair for WWUC LASER DESIG SENSOR UNIT with Base Shop ? - Jungle Int from COA2 
Repair for WWUC LASER DESIG SENSOR UNIT with Base Shop ? - Jungle Int from COA 3 


® 
x= 
@ 
G 
faa 
va 
5 
. 
o 
2 
Ee 
3 
z 





2007-1 2008-@2 2009-3 2010-04 2011-4 
Date 





Figure 9. Number of maintenance tasks performed on the laser designator unit 
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From the analysis, it is evident that COA 2 is the preferred COA. It has lower 
overall O&S costs for spares replenishment, and provides the least amount of impact on 
the repair facility. These effects are accomplished with no negative impact on Ao. 

The preceding analysis and discussion is an example of the quick-turn type of 
analysis that can be accomplished with TLCM-AT. However, it must be remembered 
that each COA had to be modeled separately in the Access database before execution of 
the simulation. Depending on the user’s familiarity with the TLCM-AT modeling 


process, this could be a fairly time-consuming task. 


c; TLCM-AT WITH DOE IMPLEMENTATION 


This section discusses the analytical process and results after simulation with 
DOE implementation and insights gained. As in the previous analysis, the focus in this 
section is directed at the Ao in quarters 12 and 20. This section contains regression 
analysis, t-Tests for two-sample means and an evaluation of the best design points. It 
should be noted here that the practical change in Ao after the simulation runs is very 


minimal. In other words, changes are too small to effect a decision. This result is in line 


with the results seen in Section B. The ranges of Ao are depicted in Table 3. 


Baseline COAI COA2 COA3 Baseline COAI 


Max Ao 82.18% 83.00% 82.83% 83.15% 90.19% 90.31% 90.11% 90.41% 
MinAo 80.41% 80.39% 81.23% 80.76% 87.43% 88.73% 88.84% 88.69% 





Table 3. Ao ranges per COA for quarters 12 and 20 


The maximum and minimum Ao in all simulation runs, across all COAs and 


design points is depicted in Table 4. 





Max Ao 83.15% Max Ao 90.41% 
Min Ao 80.39% Min Ao 87.43% 


Table 4. Ao ranges across all COAs and design points 
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iL; Regression Analysis 


Regression is a technique used for the modeling and analysis of numeric data 
consisting of values of a dependent response variable (Ao) and one or more independent 
explanatory variables (spares, induction quantity, capacity, degradation time, service 
time). Ao is the only response variable used because of time constraints to complete this 
study. The dependent variable in the regression equation is modeled as a function of the 
independent variables. Regression analysis shows the independent variables that are 
statistically significant to the dependent variable. Among many independent variables, 
only a few of them may be significant in determining the measure of interest. 

This analysis uses stepwise regression to evaluate the data obtained from the 
simulation runs of the COAs discussed in Chapter IJ. The basic procedures for stepwise 


regression involve: 


1. Identifying an initial model. 


2s Iteratively stepping, or repeatedly altering, the model at the previous step 
by adding or removing an independent variable. 


os Terminating the search when stepping is no longer possible given the 
stepping criteria or when a specified number of steps have been reached. 


The forward selection approach is used in this analysis. Forward selection 
involves starting with no variables in the model and trying out the variables one by one, 
and including them if they are statistically significant. 

Thirty-two regression models are evaluated. One model is developed for 
quarter 12 and another for quarter 20 for each scenario. Regression determines 
significant factors affecting Ao, and the variance in Ao, explained by the five varied 
factors. The amount of variance explained by the factors is represented by the coefficient 
of determination, or R’: as the R? increases, the amount of explained variance in the 
statistical model increases. A low R° indicates there could potentially be more factors 
affecting the outcome than those modeled. The R’ from all stepwise regressions is shown 


in Table 5. 
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QTR Main Effects | 2-way Interaction _3-way Interaction Quadratic 


Base Scenario 
Base Scenario 
Scenario 1 
Scenario 1 
Scenario 2 
Scenario 2 
Scenario 3 
Scenario 3 





Table 5. R’ from all stepwise regressions 


Table 5 shows relatively low R* values for scenarios 1-3, especially during 
quarter 20. Furthermore, the R” increases in all cases as interaction and polynomial 
(quadratic) terms are added to the model. The increase in R* shows that including only 
main effects does not capture the complete picture; interaction terms reveal synergies 
between the modeled factors. Practically, this shows that adjusting only one or two 
factors independently may not be the best solution if seeking to maximize Ao. 
Furthermore, this form of regression cannot adequately capture the richness of TLCM- 
AT; other factors added to the model will increase R’. In the cases where R? is low, the 
results show that other factors should be considered. 

Scenario two quarter 12 is used as an example. The following stepwise regression 
includes all two-way interaction and three-way interaction terms. Table 6 shows the 
summary of fit and parameter estimates produced from the regression are shown in 


Table 7. 














Summary of Fit 
RSquare 0.400167 
RSquare Adj 0.216545 
Root Mean Square Error 0.00252 
Mean of Response 0.822325 
Observations (or Sum Wgts) 129 
Table 6. Summary of fit for scenario 2 quarter 12 
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Parameter Estimates 
Term Estimate Std Error tRatio Probs\t| 
Intercept 0.8214981 0.001445 568.48 <.0001* 
Spares 0.0001396 2.959e-5 4.72  <.0001* 
IQ 1.4267e-5 2.811e-5 0.51 0.6129 
Capacity -0.000043 2.723e-5 -1.58 0.1176 
Degradation -0.000271 0.000799 -0.34 0.7355 
ServTime -0.000042 8.633e-5 -0.49 0.6283 
(Spares-15.5039)*(IQ-15.5039) -4.132e-7 3.015e-6 -0.14 0.8913 
(Spares- 15.5039) *(Capacity- 15.5039) -2.522e-6 4.275e-6 -0.59 0.5566 
(Spares-15.5039)*(Degradation-1.00001) 0.0002071 0.00011 1.88 0.0634 
(Spares-15.5039)*(ServTime-5.00001) 0.0000195 9.31e-6 2.09  0.0388* 
(IQ-15.5039) *(Capacity- 15.5039) -4.408e-6 3.632e-6 -1.21 0.2279 
(IQ-15.5039) *(Degradation- 1.00001) -0.000269 0.000133 -2.03 0.0452* 
(IQ-15.5039)*(ServTime-5.00001) 4.8612e-7 9.094e-6 0.05 0.9575 
(Capacity- 15.5039) *(Degradation- 1.00001) 0.0000949 8.844e-5 1.07 0.2859 
(Capacity- 15.5039) *(ServTime-5.00001) -1.661e-6 0.000012 -0.14 0.8894 
(Degradation- 1.00001)*(ServTime-5.00001) 0.0002695 0.000295 0.91 0.3628 
(Spares-15.5039)*(IQ- 15.5039) *(Capacity- 15.5039) -1.221e-7  4.666e-7 -0.26 0.7942 
(Spares-15.5039)*(IQ-15.5039)*(Degradation- 1.00001) -2.661e-5 1.363e-5 -1.95 0.0537 
(Spares-15.5039)*(IQ-15.5039)*(ServTime-5.00001) 1.7182e-6 1.112e-6 1.55 0.1255 
(Spares - 15.5039) *(Capacity- 15.5039) *(Degradation- 1.00001) 6.7712e-6 1.366e-5 0.50 0.6211 
(Spares-15.5039)*(Capacity - 15.5039) *(ServTime-5.00001) -1.348e-7 1.369e-6 -0.10 0.9218 
(Spares - 15.5039) *(Degradation-1.00001)*(ServTime-5.00001) -0.000046 3.86e-5 -1.19 0.2371 
(IQ-15.5039) *(Capacity- 15.5039) *(Degradation- 1.00001) -2.051e-6 1.385e-5 -0.15 0.8826 
(IQ-15.5039) *(Capacity- 15.5039) *(ServTime-5.00001) -1.985e-6 1.198e-6 -1.66 0.1007 
(IQ-15.5039) *(Degradation- 1.00001)*(ServTime-5.00001) -5.255e-5 3.982e-5 -1.32 0.1900 
(Capacity- 15.5039) *(Degradation-1.00001)*(ServTime-5.00001) 0.0000417 0.000039 1.07 0.2866 
(Spares-15.5039)*(Spares- 15.5039) -1.067e-5 4.502e-6 -2.37  0.0197* 
(IQ-15.5039) *(IQ- 15.5039) -5.518e-6 4.21e-6 -1.31 0.1931 
(Capacity- 15.5039) *(Capacity- 15.5039) -8.746e-7 4.012e-6 -0.22 0.8279 
(Degradation-1.00001)*(Degradation- 1.00001) 0.0050247 0.003508 1.43 0.1553 
(ServTime-5.00001)*(ServTime-5.00001) 4.5964e-5 3.744e-5 1.23 0.2225 
Table 7. Parameter estimates for scenario 2 quarter 12 


Number of spares is the dominant factor in the model. Forty percent of the 
variance is explained. Table 6 shows the summary of fit and parameter estimates 
produced from the regression are shown in Table 7. 

The significant factors in determining Ao (as shown in Table 7 with 


accompanying asterisk) are: 


a Spares 

7 Spares x Spares 

7 Spares x Service Time 

7 Induction Quantity X Degradation 
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A final model is created by removing all insignificant factors. Induction Quantity 
X Degradation is now an insignificant factor. The R’ drops from 40% to 24% of the 
variance explained by the model. Number of spares is the dominant factor in the final 
model. Table 8 shows the summary of fit and parameter estimates produced from the 


regression are shown in Table 9. 























Summary of Fit 

RSquare 0.242189 

RSquare Adj 0.224002 

Root Mean Square Error 0.002508 

Mean of Response 0.822325 

Observations (or Sum Wgts) 129 

Table 8 Final model summary of fit for scenario 2 quarter 12 
Parameter Estimates 
Term Estimate Std Error tRatio Probs|t| 
Intercept 0.8213247 0.000523 1571.2 <.0001* 
Spares 0.0001175 2611e5 4.50 <.0001* 
(Spares-15.5039)*(ServTime-5.00001)  2.2623e-5  8.165e-6 2.77  0.0064* 
(Spares- 15.5039) *(Spares- 15.5039) -1.148e-5 3.434e-6 -3.34 0.0011* 
Table 9 Final model parameter estimates for scenario 2 quarter 12 


The significant factors in determining Ao (as shown in Table 9 with 


accompanying asterisk) are: 


7 Spares 
a Spares x Spares 
a Spares x Service Time 


The coefficients are small; this indicates that these terms, while statistically 
significant in the model, have minimal practical impact on Ao. In other words, the 
increase in Ao for each spare added is too small to effect a decision. Similar conclusions 


are drawn from the other COAs as well. 


2 Two-Sample Comparison 


Following regression analysis, paired two-sample t-Tests are done between each 


scenario. The purpose of a t-Test is to determine whether or not two population means 
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are equal. The two-sample t-Test is performed on two data sets that are assumed to have 
been drawn from populations that follow a normal distribution with constant variance. 
However, for sufficiently large samples there are moderate to robust departures from 
these assumptions. The data sets used here are the Ao values generated for each scenario 


over the 129 design points. Six t-Tests are completed to cover all possible combinations: 


7 Base — Scenario 1 
7 Base — Scenario 2 
7 Base — Scenario 3 
7 Scenario 1 — Scenario 2 
7 Scenario 1 — Scenario 3 
7 Scenario 2 — Scenario 3 


In each test, the hypothesized mean difference is zero (no difference in population 
means). Tables 10 and 11 depict the results of all paired two-sample t-Tests for 


differences in Ao after quarters 12 and 20, respectively. 
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t-Test: Paired Two Sample for Means 
Quarter 12 


Hypothesized Mean Difference 
df 

P(T<=t) two-tail 

t Critical two-tail 


Hypothesized Mean Difference 
df 

P(T<=t) two-tail 

t Critical two-tail 


Hypothesized Mean Difference 
df 
P(T<=t) two-tail 


t Critical two-tail 


Observations 

Hypothesized Mean Difference 
df 

P(T<=t) two-tail 

t Critical two-tail 


Observations 

Hypothesized Mean Difference 
df 

P(T<=t) two-tail 

t Critical two-tail 


Observations 

Hypothesized Mean Difference 
df 

P(T<=t) two-tail 

t Critical two-tail 


Table 10. 


0.811409591 
1.1044E-05 
129 

0 

128 
4.65353E-33 
1.978670823 


0.818230432 
2.78253E-05 
129 

0 

128 
2.64123E-15 
1.978670823 


0.822325324 
8.1056E-06 
129 

0 

128 
0.022398628 
1.978670823 


0.818230432 
2.78253E-05 
129 

0 

128 
8.1068E-12 
1.978670823 


0.811409591 
1.1044E-05 
129 

0 

128 
3.37578E-65 
1.978670823 


0.811409591 
1.1044E-05 
129 

0 

128 
2.87256E-54 
1.978670823 


0.818230432 
2.78253E-05 
129 


1.70177E-05 
129 


1.70177E-05 
129 


1.70177E-05 
129 





t-Test results for difference in Ao for 12" quarter 


t-Test: Paired Two Sample for Means 
Quarter 20 


Hypothesized Mean Difference 
df 

P(T<=t) two-tail 

t Critical two-tail 


Observations 

Hypothesized Mean Difference 
df 

P(T<=t) two-tail 

t Critical two-tail 


Observations 

Hypothesized Mean Difference 
df 

P(T<=t) two-tail 

t Critical two-tail 


Observations 

Hypothesized Mean Difference 
df 

P(T<=t) two-tail 

t Critical two-tail 


Hypothesized Mean Difference 
df 

P(T<=t) two-tail 

t Critical two-tail 


Hypothesized Mean Difference 
df 

P(T<=t) two-tail 

t Critical two-tail 


Table 11. 


0.890403 15 
3.75235E-05 
129 

0 

128 
3.30143E-13 
1.978670823 


0.895018571 
8.24482E-06 
129 

0 

128 
0.177946263 
1.978670823 


0.895466579 
6.7266E-06 
129 

0 

128 
0.332484251 
1.978670823 


0.895018571 
8.24482E-06 
129 

0 

128 
0.745610006 
1.978670823 


0.890403 15 
3.75235E-05 
129 

0 

128 
4.25768E-14 
1.978670823 


0.890403 15 
3.75235E-05 
129 

0 

128 
1.87398E-12 
1.978670823 


0.895018571 
8.24482E-06 
129 


0.895466579 
6.7266E-06 
129 


0.895 127699 
8.15099E-06 
129 


0.895 127699 
8.15099E-06 
129 


0.895466579 
6.7266E-06 
129 


0.895 127699 
8.15099E-06 
129 





t-Test results for difference in Ao for 20" quarter 


The low p-value calculated for each comparison reveals, in all six cases, a 
statistically significant difference in the mean Ao between each scenario for quarter 12. 
However, the higher p-values evident in quarter 20, for comparison between scenarios 1, 
2, and 3, show that there is no statistically significant difference in the mean Ao. 

The statistical significance shown in quarter 12 does not necessarily imply 
practical significance. Because the mean Ao in all cases are relatively close, the mean 
difference is not practically significant; going from 81.8% to 82.1% Ao is not significant 


enough to make practical impacts on the overall operational picture. 


3. Design Point Evaluation 


Additional insight can be obtained by taking a closer look at the design points 
producing the highest Ao. The specific factor settings in the best design points can be 
used to guide a decision maker when determining the level to set for specific factors. The 
design points, with their associated factor settings, that produced the highest Ao for 
scenario two are depicted in Table 12. 


Rank Design Point # Spares Capacity Degradation ServTime 


16 
13 
Quarter 20 


1 
2 
3 
4 
5 
6 
7 
8 
9 
10 





Se mPADNARWBNHHS 


Table 12. Best design points for scenario two 
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Design points that rank in the top ten for both quarters are in bold italics in 
Table 11. As an example, if it was determined that finishing the contingency with the 
highest Ao was a priority, design point 86 (Spares: 22, Induction Quantity: 24, 
Capacity: 6, Degradation: .5469, Service Time: 5.0781) should be used as a guide to 
determine the optimal settings to use for the indicated factors. Design point 86 shows a 
high number of spares and high induction quantity provides maximum Ao. For a 
long-range look (20 quarters), design point 53 (Spares: 10, Induction Quantity: 7, 
Capacity: 18, Degradation: 1.0078, Service Time: 9.5313) should be used as a guide to 
determine the optimal settings. Design point 53 shows a lower number of spares, lower 
induction quantity and higher capacity provide maximum Ao for the full twenty quarter 
period. In this setting (design point 57) increasing the depot capacity allows a higher Ao 


while maintaining a lower level of spares and induction quantity. 
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Vv. CONCLUSIONS 


A. RESEARCH SUMMARY 


This research set out to explore the capabilities of the TLCM-AT. Through 
development of a Java program to automate implementation of DOE with TLCM-AT, 
this thesis produces a thorough experimental concept and analytical technique to explore 
the capabilities TLCM-AT. The simulated scenarios used in this analysis form a strong 
foundation for further studies of TLCM-AT, using DOE to analyze life cycles. The 
analysis produces a process for decision makers to gain insight on weapon system life 


cycles using TLCM-AT. This thesis provides the following: 


" A framework for using DOE with TLCM-AT. 
7 A structured use of TLCM-AT for USMC decision makers. 


7 Enhancements that enable more effective usage of TLCM-AT. 


B. DOE AND TLCM-AT 


TLCM-AT uses an Access database for all inputs and outputs. The Java program 
written for this study enables manipulation of the database and variation of input. The 
program development applies Java-Access interaction to change the desired elements in 
the database. This type of analysis required an intimate familiarity with TLCM-AT 
modeling logic. For example, changing induction quantity requires an update to a single 
input table; however, implementing a change in the number of spare parts requires 
updates to multiple input tables. 

When performing DOE with TLCM-AT, the Java program is essential to 
automating the process of executing multiple design point simulations. The program is 
necessary to not only execute changes to the database, but to also keep track of the 
specific factor settings for each design point and provide the output in a form amenable to 


post processing. 
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Cc. TLCM-AT OBSERVATIONS 


Several observations were made during the course of this research on effective 
use of TLCM-AT. Chapter IV shows how TLCM-AT provides a quick-turn analysis of 
specific “what-if” scenarios. This is accomplished mainly through use of TLCM-AT’s 
GUI and embedded analytical and graphing utilities. The GUI contains a scenario editor 


interface to allow for changes in the following areas: 


7 Capacity constraints 
a Component induction 


7 Depot upgrades 


7 Evacuation probabilities 

7 Induction schedules 

7 Limits and screens 

7 Logistical consequences 

7 Maintenance task times 

7 Operational tempo 

7 Parts configuration 

7 Serial number planned usage 
7 Shipment times 

. Spares management 

. Unscheduled removal rates 


TLCM-AT is a data-driven tool. The results from scenario runs are completely 
dependent on the quality of the data collected and modeled in the database such as miles 
driven, shots fired, repair times, and depot capacities. This fact demands accurate 
recording of usage data throughout the fleet enterprise of all weapons systems. To ensure 
valid results, recorded data must be accurate, timely, and modeled correctly. The 
modeling process requires a solid understanding of TLCM-AT data entry to represent 


specific scenarios. 
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The analysis uses the average Ao taken from each design point. Access to raw 
data generated from each replication, within each design point, is not available for 
analysis. Access to TLCM-AT raw output data would allow mapping into distributions 
and analysis of any outliers. 

The process of conducting many simulation runs would be greatly enhanced if 
TLCM-AT could be run using a cluster of computers. This approach was unsuccessful 
due to the inability for TLCM-AT to be run in a headless mode (without a GUI). The 
NPS SEED Center computer cluster contains compute nodes without monitors to display 
the dialogue boxes required by TLCM-AT. 

The final lesson learned concerns the extensive knowledge of the TLCM-AT 
modeling process required to simulate different scenarios. The scenarios were modeled 
by a modeler and analyst for Clockwork Solutions (Dr. Peter Figliozzi). While a basic 
knowledge of TLCM-AT’s functional utilities will allow editing of various inputs 
through the GUI, a more robust knowledge is essential when modeling different scenarios 


via the database. Scenario development should be accessible to the typical user. 


D. FOLLOW-ON RESEARCH 


This thesis serves as a template for future life-cycle studies utilizing TLCM-AT 
and DOE. Real-life scenarios, or COAs, can be modeled and many different MOEs can 
be studied. A list of possible MOEs to study in the future is shown in Table 13. 


4] 


Age is the average hours accumulated divided by average number of objects in use. 
Events include part failures, life limited, induction and repairs. 

Avvaitng maintenance (AWN) statu 
Time in AWM, probability towards availability and total number of requests. 

Awaiting parts (AWP) status Time spent in waiting queue; includes AWP status, requests and total repairs. 

Time in AWP, probability towards availability and total number of requests. 
Average LRU removals due to life limits and failures by base by quarter. 
ar isl Sev es 


Table 13. MOEs for future study 





It must be noted that varying factors other than those in this thesis or gathering 
different outputs will require additional Java coding similar to the code detailed in 
Lieutenant Commander Garcia’s thesis (to be completed in September 2008). 

Future work could also entail a study to integrate the cluster of computers into the 
TLCM-AT and DOE process. This process will require a joint effort between 


Clockwork Solutions and SEED Center research personnel. 
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low level 1 
high level 30 
decimals 0 
factorname Spares 


APPENDIX 


0.5 0 
1.5 10 
4 4 
Deg ST 


1.25 1.25 
1.4531 4.9219 
1.4219 3.75 
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low level 1 
high level 30 
decimals 0 
factorname Spares 
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